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800 U.S. schools. This study investigated the effects of the HOTS 
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thinking skills. Of secondary interest was the examination of gender 
differences in the HOTS and Chapter 1 programs. Subjects included 113 
students in the HOTS group and 72 in the traditional Chapter 1 group. 
HOTS subjects received 45 minutes of instruction each day using the 
HOTS curriculum and computer applications. Student sel f-concept , 
achievement, and thinking skills were measured before and after the 
intervention. Findings suggested that the HOTS program was effective 
in raising student self-concept and some higher order thinking skills 
in grade 5. The program appeared to be more effective after 2 years 
of treatment, with females affected more than males. Both HOTS and 
Chapter 1 raised student achievement scores, but there were no 
stat istical ly significant differences in achievement between the two 
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In 1988, it was estimated that 25 million of the nations 64 million children were at risk 
educationally when one of the following five risk factors was considered, race, poverty, family 
structure, language background, and mother's education (Mills, Stork & Krug, 1992). It is no 



longer enough to try to improve on what schools are already doing, we must now change school 



A large majority of these children who are at risk educationally pass through the largest 
federal compensatory education program in the United States, Chapter 1 . The Chapter 1 program 
was started 30 years ago and now serves approximately 5 million children annually (Anderson, 
Hollinger & Conaty, 1992). During its early years Chapter 1 led the way in innovative 
educational methods, but has since stagnated. The vast majority of Chapter 1 programs have 
never changed from their instructional strategies of drill and practice on basic sldlls (Asher, 1987). 
Today, many theorists, educators and researchers are calling for a change in the Chapter 1 
programs (Asher & Flaxman, 1988), 

One way of changing instruction for students who are at risk educationally is through the 
use of computers. Computers hold the promise of being able to transform today’s curriculum and 
instructional strategies instead of simply improving what schools are already doing (David, 1992). 
Computers are being used to provide active learning, authentic tasks, challenging work, complex 
problem solving, and higher order thinking skills (Dalbey & Linn, 1985; Dalton & Goodrum, 

1991; David, 1992; Kagan, 1988;Pogrow, 1987). 

Along with the use of computers to help at-risk students, educators are beginning to 
discover the benefits of teaching higher order thinking skills. Students can learn the thinking skills 
that underlie all learning and transfer these skills back to the regular content areas raising their 
achievement levels. Through teacher modeling, practice, and effective teacher questioning. 
Chapter 1 students can profit from instruction on thinking skills (Adams, 198'^; Bruno & Allen, 
1992; dcBono, 1983;Pogrow, 1987, 1988, 1990). 
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practices fundamentally in order to help these educationally disadvantaged children (David, 1992). 
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Higher order thinking skills are thinking processes that require more than simple recall of 
facts (de Bono, 1983). Thinking skills such as problem solving, decision making, conceptualizing, 
evaluating, synthesizing, and creative thinking require students to evaluate, plan, and monitor their 
thinking continuously (Lipman, 1991). It would be impossible to solve problems without 
evaluating what you know about the situation, planning how to get more information and what 
strategies to use, monitoring your strategies, and evaluating the results. 

The use of computers to enhance thinking skills instruction has proven to be effective 
(Dalton & Goodmm, 1991; Norton & Resta, 1986; Sinatra, Beaudry, Pizzo & Geisert, 1994). 
Computers may be the ideal mechanism for teaching higher order thinking skills (Saloman, 1986). 
Combining computers and thinking skills instruction with educationally at-risk students may help 
to raise Chapter 1 eligible students' achievement levels up to those of their peers, thus fulfilling the 
original intent of the Chapter 1 program. 

An example of a program that combines teaching higher order thinking and the use of 
computers is The Higher Order Thinking SkiUs program (HOTS) developed by Stanley Pogrow at 
the University of Arizona. There are now over 800 schools with HOTS programs in operation in 
the United States (Pogrow, 1990). However, there have been no empirical studies with control 
groups conducted to assess the impact of the HOTS program. Specifically, no studies have been 
conducted to compare the effects of the HOTS program on Chapter 1 eligible students in the 
fourth and fifth grades to students in the more traditional Chapter 1 programs. These Chapter 1 
programs are dominated by basic skills instruction (U.S. Department of Education, 1993b). 

The purpose of the present study was to investigate the effects of the HOTS program 
versus the traditional Chapter 1 program on fourth and fifth grade students' self-concept, reading 
achievement, and higher order thinking skills. Of secondary interest was the examination of 
gender differences in both the Higher Order Thinking Skills program and the Chapter 1 program. 
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Assumptions 

The assumptions of this investigation were: 

1 . Students selected for this study form a representative sample of Chapter 1 
eligible students in the fourth and fifth grades in the selected school district. 

2. Higher order thinking skills can be taught and learned. 

3. The .SRlf-Dftscription Questionnaire - 1 (Marsh. 1988) was a reliable and valid 

instrument for measuring students' self-concepts. 

4. The Tnwa Test of Rasir. Skills (ITBS) was a reliable and valid instrument for measuring 

students' reading achievement. 

5. The Test of Higher Cognitive Processes (Ross & Ross, 1976) was a reliable and 
valid instrument for measuring higher order thinking skills in the areas of abstract relations, 
sequential synthesis, and analysis of attributes. 

6. Test conditions were adequate for each participant. 

7. The subjects were motivated to participate. 

8. The subjects responded openly and honestly. 

9. The obtained data were reliable. 

Limitations 

The limitations of this investigation were: 

1 . The small sample size may reduce statistical power. 

2. Subjects who met the criteria for inclusion in the study were selected from one 
school district. The non-random method of selection limited generalization of the 
results of this study. 

3 The investigator was not able to control for the influence of outside experiences among 
the subject 

Overview of Methodology 

In order to examine the effects of the HOTS program versus the traditional Chapter 1 
program on fourth and fifth grade students' self-concept, achievement, and higher order thinking 
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skills, a pretest-posttest contrast group design was utilized. The subjects included 175 fourth and 
fifth grade Chapter 1 eligible students, 1 13 students in the experimental Higher Order Thinking 
Skills group and 72 students in the Chapter 1 group. 

Thft Chapter 1 program Chapter 1 students individually are characterized by free lunch 
program participation, having low family income and low educational attainment of parents (U. S. 
Department of Education, 1993a). Racial minorities are also more likely to participate in Chapter 
1 as compared to their representation in elementary schools in general (U. S. Department of 
Education, 1993a). According to state reports, 41 % of all Chapter 1 participants are White, 28 
% are Black, and 27 % are Hispanic (Sinclair, & Gutmann, 1993). 

Assignment to Chapter 1 is based primarily on a student's standardized achievement scores. 
Students are ranked by school and grade and the lowest scoring students receive services first. 

Not all students who are eligible receive services, and students do not have to fit into the above 
characteristics to qualify. 

Various types of services are typically provided through Chapter 1 including remediation for 
reading, math, and language arts. Chapter 1 classes are 45 minutes per day and include from five 
to ten students. Seventy-four percent of elementary Chapter 1 programs are pullout arrangements 
(U.S. Department of Education, 1993a). Observations and interviews of Chapter 1 classrooms, 
teachers, and students by this researcher indicated that the activities consisted mainly of drill and 
practice over remedial tasks. This is in line with Chapter 1 services across the country where 84 
% of elementary schools reported that basic skills drill and practice was a major focus of Chapter 
1 instruction (U.S. Department of Education, 1993a). Some of the tasks involved in fourth and 
fifth grade Chapter 1 sites included reading and recording the titles of below grade level books, 
language arts work sheets for below grade level skills, computer drill and practice programs on 
below grade level skills, and spelling practice with below grade level words. There was some 
project work involved in the Chapter 1 classes such as book reports or projects on books (again 
below grade level books), and research reports. However, observation of Chapter 1 classes by 
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this researcher indicated that 92 % of class time was spent on drill and practice of below grade 
level skills. 

The Higher Ordpr Thinking Skills program. All HOTS programs are of the pullout variety 
and involve students in 45 minutes of instruction per day. Each HOTS program class contained 
five to ten students. Each class progresses through three curriculum books used only by the 
HOTS teacher. Students are assigned to the Higher Order Thinking Skills program by Chapter 1 
eligibility. The HOTS program is basically designed to be substituted for other Chapter 1 
services. 

The researcher conducted observations of HOTS classrooms and interviews with students 
and teachers in these same classrooms. These observations and interviews indicated that the 
HOTS teachers were closely following the HOTS curriculum guides. These teaching guides 
provide teachers with step-by-step procedures for each lesson. The basic HOTS lesson involved a 
group discussion at the beginning of the class period. This discussion involved the teacher posing 
questions and probing students for answers. These questions and answers led to the discussion of 
a concept to be covered that day. This discussion generally lasted about 10 minutes. The 
students then spent approximately 20 minutes working at the computers on software that was 
intended to let students further develop the concept. There was an occasional activity that linked 
what was happening in the regular classroom to the HOTS curriculum. For example, students 
used their regular classroom spelling list to make crossword puzzles in HOTS. In the fifth grade 
classes there was more of a focus on writing on the computer than in the fourth grade. 

Subjects. The subjects for this study were Chapter 1 eligible students. In the state of 
Georgia there are approximately 160,000 students. Approximately 55% of these students are 
male and 45 % female. The racial make up consists of 62 % Black, 36 % White, and 2% Hispanic 
(Georgia Department of Education, 1994). In the school district used in the present study there 
are approximately 950 elementary Chapter 1 students. Of those students 81 % are Black, 16% 
are White, and 2% are Hispanic, Approximately, 55% of these students are male and 45 % 
female. More specifically, in the fourth and fifth grade in the school district there are 442 
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students, 52% male and 48% female. These students consist of 85 % Black, 1 3 % White, and 1 
% Hispanic ("Clarke County Board of Education, 1995). 

The final sample consisted of 1 75 students in the fourth and fifth grades from one school 
district in Northeast Georgia. There were 54 first year HOTS students in fourth grade classrooms 
and 49 second year HOTS students in fifth grade classrooms. The control group consisted of 35 
fourth grade and 37 fifth grade students. The subjects included 149 Blacks, 24 Whites, and 2 
Hispanic students. There were 92 female and 83 male students. 

Hata rnllpiotinn Three instruments were used to assess the effectiveness of the treatment 
programs. Student self-concept was measured by using the Self-Description Questionnaire - 1 
r SDOn (Marsh, 1988). The Towa Test of Basic Skills (UBS.) (Riverside Publishing Company, 
1986) was used to measure student reading achievement. The Ross . Test - Of H ig her Co g n i tive 
Processes (Ross & Ross, 1976) was used to measure student higher order thinking skills. 

Scores from the spring 1994 administration of the UBS were used as a pretest for the 1994- 
95 school year. The pretests for the SDQ.L and the Ross Test of Higher Cognitive Processes were 
given at the beginning of the 1994-1995 school year and were administered by this researcher. 

The posttests for all instruments were administered in the spring of 1995. The UBS test was 
administered by the individual Chapter 1 and HOTS teachers as prescribed by Chapter 1 
regulations. While the snoi and the Ross Test of Higher Cognitive Processes were administered 

by this researcher. 

In order to get as complete a picture as possible as to what the instruction was typically like 
in both the Chapter 1 and HOTS classes the researcher conducted observations at each site on a 
random selection of classes. At least two classes per site were observed during the fall, winter, 
and the spring of the school year for a minimum of 42 observations. Also to get more information 
as to the effects of the HOTS program and the more traditional Chapter 1 classes, the researcher 
conducted open-ended interviews with randomly selected students and teachers across the 
research sites. There were at least two, half - hour interviews with each selected participant, one 
in the first quarter of the year and one in the last quarter of the year. 
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Design and Analyses The present study Utilized a pretest-posttest contrast group data 
collection design (Campell & Stanley, 1963). 

Experimental group 0| O 2 O 3 Xj 0| O 2 O 3 O 4 



0 2 = Self Description Questionnaire 1 

03 = Ross Test of Higher Cognitive Processes 

0 4 = Interviews and Observations (continuous) 

X j = Higher Order Thinking Skills Program 

X 2 = Traditional Chapter I Program 

Posttest scores were compared using a two-way analysis of covariance (ANCOVA). The 
posttest scores served as the dependent measures, and the pretest scores served as the covariates. 
The following three dependent variables were examined: self-concept, reading achievement, and 
higher order thinking skills. A separate ANCOVA was conducted on each of the dependent 
variables. 

The multisite interviews and observations were analyzed using the analytic induction 
method of qualitative data analysis for triangulation data. This method was chosen because 
"analytic induction is employed when some specific problem, question, or issue becomes the focus 
of research. Data are collected and analyzed to develop a descriptive model that encompasses all 
cases of the phenomena" (Bogdan & Biklen, 1992, p. 70). This data was used to give a richer 
overall picture of the two programs. Selection from the interviews were used to help explain the 
quantitative findings. 

Null Hypotheses 

The following research hypotheses were- studied in this investigation. Each hypothesis will be 
examined separately by grade. 



Control group 



0| O2 O3 X2 Oj O2 O3 O4 



Where 
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NiiH hypothesis 1. There are no statistically significant posttest mean differences in the 
adjusted reported self-concept scores of students receiving the Higher Order Thinking Skills 
program and subjects receiving Chapter 1 services. 

Null hvnothe.sis 2. There are no statistically significant posttest mean differences in the 
adjusted reported reading achievement scores of students receiving the Higher Order Thinking 
Skills program and subjects receiving Chapter 1 services. 

Null hvnnthesis 3 There are no statistically significant posttest mean differences in the 
adjusted reported higher order thinking skill - abstract relations scores of students receiving the 
Higher Order Thinking Skills program and subjects receiving Chapter 1 services. 

Null hypothesis 4. There are no statistically significant posttest mean differences in the 
adjusted reported higher order thinking skill - sequential synthesis scores of students receiving the 
Higher Order Thinking Skills program and subjects receiving Chapter 1 services. 

>Jiill hvnnthesis 5. There are no statistically significant posttest mean differences in the 
adjusted reported higher order thinking skill -analysis of attributes scores of students receiving the 
Higher Order Thinking Skills program and subjects receiving Chapter 1 services. 

Null hvnothftsis 6. There are no statistically significant posttest mean differences in the 
adjusted reported scores on each dependent measure between female and male students receiving 
the Higher Order Thinking Skills program and subjects receiving Chapter 1 services. 

Fxamination of Null Hypotheses 

In order to examine the effects of a HOTS program on Chapter 1 eligible fourth and fifth 
grade students' self-concept, achievement, and higher order thinking skills, a pretest -posttest 
contrast group design was employed (Campbell & Stanley, 1963). Because randomization was 
not possible, the analysis of covariance procedure was used to adjust for initial group differences. 
A t-test on each of the dependent measures indicated significant preassessment differences on 
achievement and higher order thinking skill - abstract relations. Analysis of covariance 
procedures were also utilized to reduce error variance and increase statistical power. Hypotheses 
were tested using a priori alpha level of .05. 
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An assumption of the analysis of covariance is equality of regression slopes. A test for 
equality of slopes was therefore conducted for each of the three dependent measures at 
posttesting. Since the initial tests indicated that all of the regression slopes were equal at 
posttesting (all E values were nonsignificant), the analysis of covariance was then conducted to 
test each null hypothesis. 

The results of the analyses and their relationship to each of the six null hypotheses follow. 
Each hypothesis will be addressed separately by grade. The students in both fourth grade groups 
had completed one year of the program and students in both fifth grade "roups had completed 
two years of the program. Consequently, it was not possible to calculate overall changes. 

Null Hypothesis 1 . There are no statistically significant posttest mean differences in the 
adjusted reported self-concept scores of students receiving the Higher Order Thinking Skills 
program and subjects receiving Chapter 1 services. 

The obtained and adjusted means and standard deviations for both treatment groups for 
self-concept scores are summarized in Table 1. The table shows that fifth graders in the HOTS 
program had higher self-concept scores on the pretest than did fi3urth graders. This is due to the 
students being a year older and already having completed one year of the HOTS program. To 
investigate this hypothesis, a two-way analysis of covariance (ANCOVA) was performed to check 
for interactions between treatment condition and gender. No statistically sigmficant interaction 
was found. The dependent measure was next analyzed without concern about potential sources 
of interactions. The ANCOVA results in table 2 were not significant for grade four but did 
indicate a statistically significant difference between fifth grade students in the HOTS programs 
and students in Chapter 1, F (1, 83) = 3.46, p <.05. The adjusted means, as shown in Table 1, 
show that the fifth grade students in the HOTS programs had significantly higher overall self- 
concepts (M = 34.07) than students in the Chapter 1 program (M = 32.67). Therefore, null 
hypothesis 1 was rejected. 




Table 1 

A Summary of Means and Standard Deviat ions for Self-Concept bv Grade 
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HOTS 


Pre Assessment 


Post Assessment 


Adjusted 


Means 


SD 


Means 


SD 


Means 




Fourth Grade (n = 54) 


31.54 


5.66 


32.43 


5.59 


32.70 




Fifth Grade (n = 49) 


33.00 


5.49 


34.24 


3.97 


34.07 




Chanter 1 














Fourth Grade (n = 35) 


32.38 


3.74 


31.74 


3.77 


31.47 




Fifth Grade (n = 37) 


32.30 


4.17 


32.50 


4.42 


32.67 




Table 2 














Analvsis of Covariance for Students' Self-Concent bv Grade Level 






















Source df 




SS 


MS 


F 




P 


Fourth Grade 














Treatment 1 




32.01 


32.01 


2.22 




0.14 


Covariate 1 




875.51 


875.51 


60.65 




0.000* 


Error 86 




1241.47 


14.44 








Fifth Grade 














Treatment 1 




41.63 


41.63 


3.46 




0.04* 


Covariate 1 




484.79 


484.79 


40.27 




0.000* 


Error 83 




999.15 


12.04 









*| 1<05 




il 



11 



Null Hypothesis 2. There are no statistically significant posttest mean diiferences in the 
adjusted reported reading achievement scores of students receiving the Higher Order Thinking 
Skills program and subjects receiving Chapter 1 services. 

The obtained and adjusted means and standard deviations for both treatment groups for 
reading achievement scores are summarized in Table 3. The table shows that fifth graders had 
higher reading achievement scores on the pretest than did fourth graders. This is due to the 
students being a year older and already ..aving completed one year of the HOTS program. To 
investigate this question, a two-way ANCOVA was performed to check for interactions between 
treatment condition and gender. No statistically significant interaction was found. The 
dependent measure was next analyzed without concern about potential sources of interactions. 
The results from the ANCOVA are summarized in Table 4. The results indicate no statistically 
Table 3 

A Summary of Means and Standard Deviatio n s for Re ading Ach i evement by Grade 

Pre Assessment Post Assesiiment Adjusted 



HOTS 


Means 


SD Means 


SD 


Means 


Fourth Grade (n = 54) 


21.80 


11.85 33.30 


13.87 


32.40 


Fifth Grade (n = 49) 


23.25 


11.39 35.06 


13.33 


33.97 


Chapter .1 










Fourth Grade (n = 35) 


15.83 


11.53 30.43 


14.52 


31.47 


Fifth Grade (n = 37) 


16.03 


11.44 29.46 


11.02 


32.67 



significant difference in fourth or fifth grade between students in the HOTS programs and 
students in Chapter 1, Fourth Grade, F (1, 86) = 0. 13; Fifth Grade, F (1, 83) = 1.55, p >.05. The 
adjusted means, as shown in Table 3, indicate that the students in the HOTS programs had scores 
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that were not significantly higher on the ITBS (Fourth Grade, M — 32.40; Fifth Grade, M 
33.97) as compared with students in the Chapter 1 program (Fourth Grade, M — 3 1.47, Fifth 
Grade, M = 32.67). Therefore, null hypothesis 2 was accepted. 

Table 4 

Analysis of Covariance for Students' Reading A chievement by (jrade 



Source 
Fourth Grade 


df 


SS 


MS 


F 


P 


Treatment 


1 


23.76 


23.76 


0.13 


0.72 


Covariate 


1 


1212.15 


1212.15 


6.39 


0.01* 


Error 


86 


16302.57 


189.57 






Fifth Grade 












Treatment 


1 


222.77 


222.77 


1.55 


0.22 


Covariate 


1 


1447.15 


1447.15 


10.09 


0.00* 


Error 


83 


11899.62 


143.37 







*p<.05 

Null Hypothesis 3 There are no statistically significant posttest mean differences in the 
adjusted reported higher order thinking skill - abstract relations scores of students receiving the 
Higher Order Thinking Skills program and subjects receiving Chapter 1 services. 

The obtained and adjusted means and standard deviations for both treatment groups for 
achievement scores are summarized in Table 5. The table shows that fifth graders had higher 
achievement scores on the pretest than did fourth graders. This is due to the students being a year 
older and already having completed one year of the HOTS program. To investigate this 
hypothesis, a two-way ANCOVA was performed to check for interactions between treatment 
condition and gender. No statistically significant interaction was found. The dependent measure 
was next analyzed without concern about potential sources of interactions. The ANCOVA 

ERIC 
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Table 5 

A Summary of Means and Stand a rd Deviations for Higher. Order Th i nk i ng Sk ill Abstract 
Relations by Grade 

Pre Assessment Post Assessment Adjusted 



HOTS 


Means 


SD 


Means 


SD 


Means 


A X 

Fourth Grade (n = 54) 


3.57 


2.54 


3.72 


2.41 


3.85 


Fifth Grade (n = 49) 


5.39 


2.69 


5.49 


2.97 


5.75 


Chanter 1 












-1. 

Fourth Grade (n = 35) 


4.37 


3.07 


3.31 


2.35 


3.19 


Fifth Grade (n = 37) 


6.89 


3.57 


5.70 


3.52 


5.44 



results shown in table 6 indicate no statistically significant difference between students in HOTS 
programs and students in Chapter 1, Fourth Grade. F (1, 86) = 1.84; Fifth Grade, F (1, 83) — 

0.20, p >.05. The adjusted means, as shown in Table 5, indicate that the students in the HOTS 
programs had scores that were not sigmficantly higher on abstract relations (Fourth Grade, M — 
3.85; Fifth Grade, M = 5.75) as compared with students in the Chapter 1 program (Fourth Grade, 
M = 3. 19; Fifth Grade, M = 5.44, SD = 3.52). Therefore, null hypothesis 3 was accepted. 

Null Hypothesis 4. There are no statistically significant posttest mean differences in the 
adjusted reported higher order thinking skill - sequential synthesis scores of students receiving the 
Higher Order Thinking Skills program and subjects receiving Chapter 1 services. 

The obtained and adjusted means and standard deviations for both treatment groups for 
achievement scores are summarized in Table 7. The table shows that fifth graders had higher 
achievement scores on the pretest than did fourth graders. This is due to the students being a year 
older and already having completed one year of the HOTS program. To investigate this 
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Table 6 

Analysis nf Covananrp for Students' Higher Order T hinking Skill - Abstract Rela t ions b’ Grade 



Source 
Fourth Grade 


df 


SS 


MS 


F 




Treatment 


1 


9.18 


9.18 


1.84 


0.18 


Covariate 


1 


62.65 


62.65 


12.59 


0.00* 


Error 


86 


428.08 


4.98 






Fifth Grade 












Treatment 


1 


1.84 


1.84 


0.20 


0.66 


Covariate 


1 


94.07 


94.07 


10.08 


0.00* 


Error 


83 


775.02 


9.34 







*Il<.05 
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hypothesis, a two-way ANCOVA was performed to check for interactions between treatment 
condition and gender. No statistically significant interaction was found. The dependent measure 
was next analyzed without concern about potential sources of interactions. The results of the 
ANCOVA by grade in table 8 were not significant for fourth graders. However, a statistically 



Table 7 



A Summary of Means and Standard Deviations. 




Skill - Sequential 



Synthesis by Grade 



Pre Assessment Post Assessment Adjusted 



Means 



HOTS 

Fourth Grade (n=54) 0.83 

Fifth Grade (n = 49) 1.12 

Chapter 1 

Fourth Grade (n = 35) 0.77 

Fifth Grade (n = 37) 0.97 



SD Means SD Means 



0.95 


1.11 


1.18 


1.11 


1.09 


1.78 


1.19 


1.78 


0.88 


0.74 


0.85 


0.74 


1.14 


0.97 


1.01 


0.97 



significant difference in fifth grade between students in the Higher Order Thinking Skills 
(HOTS) programs and students in Chapter 1, F (1, 83) = 10.52, p <.05 was found. The obtained 
adjusted means, as shown in Table 7, indicates that the fifth grade students in the HOTS programs 
had significantly higher overall sequential synthesis scores (M =1.78) as compared with students 
in the Chapter 1 program (M = 0.97). Therefore, null hypothesis 4 was rejected. 
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Table 8 

Analysis nf Tnyariance for Students' Higher O rder Thinking Skill - Sl^Quent i al Synthesis bv Gra de 



Source 
Fourth Grade 


df 


SS 


MS 


F 




Treatment 


1 


3.00 


3.00 


2.66 


0.11 


Covariate 


1 


1.11 


1.11 


0.99 


0.32 


Error 


86 


96.91 


1.13 






Fifth Grade 












Treatment 


1 


13.36 


13.36 


10.52 


0.00* 


Covariate 


1 


0.11 


0.11 


0.08 


0.77 


Error 


83 


105.40 


1.27 







*p<.05 

Null Hypothesis S There are no statistically significant posttest mean differences in the 
adjusted reported higher order thinking skill -analysis of attributes scores of students receiving the 
Higher Order Thinking Skills program and subjects receiving Chapter 1 services. 

The obtained and adjusted means and standard deviations for both treatment groups for 
higher order thinking skill - analysis of attributes scores are summarized in Table 9. The pretest 
means are higher for fifth graders in the HOTS program, reflecting one more year of treatment 
and a year older. To investigate this hypothesis, a two-way ANCOVA was performed to check 
for interactions between treatment condition and gender. No statistically significant interaction 
was found. The dependent measure was next analyzed without concern about potential sources 
of interactions. 
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Table 9 

A Summary of Means and Standard Deviations for H ig her Order T hitlkinfi . Sk i ll - Analys i s of 
Attributes by Grade 



Pre Assessment Post Assessment Adjusted 



Means SD Means SD Means 



HOTS 

Fourth Grade (n = 54) 
Fifth Grade (n = 49) 
Chanter 1 


7.50 

7.88 


2.30 

2.98 


7.46 

8.45 


1.94 

1.86 


7.48 

8.44 


Fourth Grade (n = 35) 


7.94 


1.59 


7.43 


1.75 


7.41 


Fifth Grade (n = 37) 


7.59 


1.61 


7.46 


1.98 


7.47 



The results of the ANCOVA by grade were not significant for fourth graders, but were 
significant for fifth grade students, F ( 1, 83) = 5.34, p < .05. The adjusted means in Table 9 
indicate that the fifth grade students in the HOTS program had significantly higher scores for 
sequential synthesis (M = 8.44) as compared with fifth grade students in the Chapter 1 program 
(M = 7.47). Therefore, null hypothesis 5 was rejected. The results from the ANCOVA are 
summarized in Table 10. 

Null Hypothesis 6 There are no statistically significant posttest mean differences in the 
adjusted scores on each dependent measure between female and male students receiving the 
Higher Order Thinking Skills program and subjects receiving Chapter 1 services. 

Readers should remember that to investigate this question, two-way ANCOVA were 
performed on each dependent measure to check for interactions between treatment condition and 
gender. No statistically significant interaction were found. The dependent measures were next 
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Table 10 

Analysis nf rnvariance for fstiidents' H ipher Ord er Thinking Sk i ll - Analysis of Attributes b y: 



Grade 


Source 
Fmirth firade 


df 


SS 


MS 


F 


P 


Treatment 


1 


0.08 


0.08 


0.02 


0.88 


Covariate 


1 


1.49 


1.49 


0.42 


0.52 


Error 


86 


302.51 


3.52 






Fifth Grade 












Treatment 


1 


19.45 


19.45 


5.34 


0.02* 


Covariate 


1 


4.92 


4.92 


1.35 


0.25 


Error 


83 


302.39 


3.64 







*ji<.05 

analyzed without concern about potential sources of interactions. There were no statistically 
significant differences ft and between male and female scores on the dependent measures of self- 
concept, achievement, and higher order thinking skills - sequential synthesis and analysis of 
attributes. However, there was statistical significance found, for fifth graders in the HOTS 
program, in higher order thinking skill - abstract relations, F (1, 46) = 8.90, p <.05 as shown m 
tablel2. The means, as shown in Table 1 1, indicate that fifth grade female students in the HOTS 
program had significantly higher scores as compared with fifth grade male students in the HOTS 
program. Therefore, null hypothesis 6 was rejected. 
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Table 11 

A Summary of Means for Higher Order T hinking Skill - Abstract Relations by Ge nd er . 



Pre Assessment 


Post Assessment 


Adjusted 


Means SD 


Means SD 


Means 



HOTS 

Fourth Grade (n = 54) 



Male (n = 27) 


3.26 


2.77 


3.19 


2.24 


3.28 


Female (n = 27) 


3.89 


2.29 


4.26 


2.51 


4.16 


Fifth Grade (n = 49) 












Male (n = 23) 


6.09 


2.47 


4.43 


2.63 


4.23 


Female (n = 26) 


4.77 


2.78 


6.42 


3.00 


6.63 



Chapter 1 

Fourth Grade (n = 35) 



Male (n = 16) 


2.88 


3.12 


2.81 


2.26 


3.22 


Female (n=19) 


5.63 


2.45 


3.74 


2.40 


3.33 


Fifth Grade (n = 37) 












Male (n = 17) 


6.12 


3.74 


4.88 


3.28 


5.18 


Female (n = 20) 


7.55 


3.36 


6.40 


3.65 


6.10 
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Table 12 

Analysis r>f rnvariance for Students' Higher O rder Thinking Skill - Abstract Relat i ons fo i 
Students in the HOTS P rogram hv Gender 



Source 
Fourth Grade 


df 


SS 


MS 


F 


R 


Gender 


1 


10.21 


10.21 


2.00 


0.16 


Covariate 


1 


38.50 


38.50 


7.55 


0.00* 


Error 


51 


260.13 


5.10 






Fifth Grade 












Gender 


1 


66.38 


66.38 


8.90 


0.01* 


Covariate 


1 


32.92 


32.92 


4.41 


0.04* 


Error 


46 


343.08 


7.46 







*p<.05 

Discussion of the Results 

Results of the study for each of the dependent variables are discussed in the following 
sections. Analyses were conducted separately by grade level. 

Self-Concept For null hypothesis one the effect of training on students' self-concept as 
measured by the Self-Descriptive Questionnaire 1 was examined. An analysis of covariace was 
utilized to test the null hypothesis with individual pretest scores serving as the covariate. The 
results indicated that there was a statistically significant difference in the self-concept of fifth 
grade students in the two treatments. Fifth grade students in the Higher Order Thinking Skills 
program had significantly higher self-concepts after treatment than did students in the Chapter 1 
program. There was no significant difference between the treatment groups on self-concept 
scores at the fourth grade, however. No significant interactions were found between the 
treatment received and whether the student was male or female. 
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One possible explanation for the positive finding in fifth grade could be the effect of 
working daily with computers. Interviews with students from both treatment groups indicated 
that students viewed computers as activities for "smart" students. A common response to 
questions about who gets to work on computers was "Only kids that go to Spectrum (gifted 
classes) get to work on computers" or "We get to work on computers one time a week. They 
(spectrum students) work on them (computers) every day." Students in the HOTS program made 
statements about how they felt on "Visitation days" when they got to bring someone to HOTS to 
show what they could do on the computers; "They thinlc we are real smart 'cause they can't do the 
computers like we can" ; "My mom came and she was surprised that I could do better than her on 
the computer." and "I beat my teacher on Carman Sandiago." Working daily with the computers 
may help to raise students' self-concept. 

Students perceptions of how their peers and parents view the two treatment programs may 
have influenced the rise in students' self-concepts. Students in the Chapter 1 program made 
statements about how others think about Chapter 1 such as "They know we do work that little 
kids do" ; "Mom said if I learn to read good I don't have to go to that class (Chapter 1) any more"; 

"My friends say I go to the dumb class" ; and "Nobody wants to go to Mrs class. We do 

baby things." At the same time the Higher Order Thinking SkiUs program seems to be viewed 

more positively; " and (students friends) keep asking when they get to go to 

HOTS" ; "Everyone keeps on saying why do we get to do all that fiin stuff in HOTS , and They 
(classmates) always want me to help them when we go to computer lab because I work with the 
computers all the time." Peers and parents views of the programs may have an influence on 
students' self-concept. 

Another possible explanation for this finding could be the program itself. Students higher 
self-concept may stem from different teaching methods in the Higher Order Thinking Skills 
program. Teaching techniques such as "controlled floundering" force students to work through 
difficult problems on their otvn. Students are allowed to initially fail at an activity then given the 
time and encouragement needed to work through to a successful solution. Students in the HOTS 
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program talkod in interviews about having to work through problems on their own and the lack of 
"help" from the HOTS teacher. Students made statements such as "She (the teacher) won't help 
you" ; "The teacher just says you can do it and everything you need to solve the problem is on the 
screen." Stopping teachers from giving unnecessary help v'hen students encounter difficulties is 
part of the training for HOTS teachers. According to interviews with the teachers, "It is hard to 
let students get frustrated and not help them, but when all the information is on the screen they 
seem to learn better when they work it out themselves." The differences in treatment may 
contribute to this finding. 

The lack of any significant difference in fourth grade may be due to a lack of time in the 
HOTS treatment group. The HOTS developers suggest it may take more than one year to find 
positive results from the program and this finding seems to support this claim. The absents of 
significant difference betw,een males and females may be due to the age of the students. 

Differences in self-concepts between males and females may not manifest themselves until later in 
adolescents. 

Reading Achievement There was no statistically significant difference found in grade four 
or five between the adjusted post treatment means for reading achievement. There was also no 
statistically significant difference found between male and female adjusted post treatment means 
for reading achievement. On the average both treatment programs were effective in raising the 
participants achievement scores. The average score for a student in the Higher Order Thinking 
Skills programs was raised 1 1 .50 NCEs in fourth grade and 1 1 .82 NCEs in fifth grade. While the 
average score for a student in the Chapter 1 program was raised 14.60 NCEs in fourth grade and 
13.43 in fifth grade. 

One possible explanation may be that the increase in skills learned in the HOTS program 
have not had enough time to transfer to an increase in reading achievement on standardized test. 
The students may need more time in the program or more coordination between the Higher Order 
Thinking Skills program and the regular classroom. 
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Both teachers and students in the HOTS program talked about the need for regular 
classroom teachers to use strategies employed in the program. HOTS teachers made statements 
such as "We need to provide their regular teachers some training like we got so that they can use 
the same strategies in the kid's rooms" ; "I think the students need this type of teaching all day 
long. The way we question students and have them work through problems really helps them. 
Students also talked about the questioning or probing HOTS teachers do in class. "She (HOTS 
teacher) asks us questions about the word and we have to figure out the word. She will like. ..she 
can't give us clues so she just stays on us and stays on us. That is what is called probing. She 
don't go on to the next person right away"; "When you're being probed it seems like you're never 
going to get the answer. Why is she doing this to me? And then when you get it you get so 
excited"; "When you get the answer everyone claps for you"; "I try not to give up and she tells us 
that we can not give up and we have to do it." When asked, HOTS students said they wished 
their classroom teacher would ask questions the way thdr HOTS teacher did. "If you don't know 

the answer in Mrs. class (the regular classroom teacher) she calls on someone else or tells 

you" ; "In my class I just sit there 'cause the teacher will go on (to another student)"; "In class (the 
regular classroom) I feel like I don't know the word because my teacher just rold me the answer 
and I don't know nothing about the word." It is possible that the use of the HOTS strategies in 
the regular classroom would help to facilitate the transfer of skills. 

Higher Order Thinking Skills - Abstract Relations. There was no statistically sigmficant 
difference found in fourth or fifth grade between the adjusted post treatment means for abstract 
relations. Students in the Chapter 1 program on the average actually received lower scores on the 
posttest than on the pretest. This regression could be due to the Chapter 1 programs 
concentration on basic skills. Students may become so used to short easy answers that they fail to 
try on harder problems, spend less time working for an answer, and resort to guessing more often. 
This could result in a reduction in the average score on higher order thinldng skills. 

One possible explanation for the lack of significant gains in this higher order thinking skill 
may be that the two programs might raise some skills but not others. There is a great debate over 
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what are higher order thinking skills, how are they organized, and how best to teach students to 
use these skills. It is very possible that it requires many types of instruction in many ways in order 
to teach a variety of thinking skills, and that these two programs are not the ideal way to provide 
instruction for this particular thinking skill. 

There were, however, statistically significant differences found for abstract relations 
between males and females in the HOTS program. Fifth grade female students in the HOTS 
program had statistically higher scores than fifth grade males in the HOTS program. This finding 
is supported by student interviews. Female students, on average, were able to express what they 
had learned in the program better than male students. One possible explanation could be that, 
developmentally, female students in the fourth and fifth grade may be more effected by the HOTS 
program. 

Synthesis The results of this study indicated a statistically significant 
difference in the higher order thinking skill of sequential synthesis of the students in fifth grade in 
the two treatments. Fifth grade students in the HOTS program had sigmficantly higher scores on 
sequential synthesis than fifth grade students in Chapter 1. There was no significant difference 
found between the treatment groups at the fourth grade or between males and females. It is 
possible that the HOTS program does a better job raising student sequential synthesis skills, but it 
takes more than one year for the difference to manifest itself This finding would support the 
claims by the HOTS promoters that it takes two years of training in the program for results to be 
seen. 

Analysis of Attributes The results of this Study indicated a statistically significant 
difference in the higher order thinking skill - analysis of attributes of fifth grade students in the 
two treatment groups. Fifth grade students in the HOTS program had sigmficantly higher scores 
on the higher order thinking skill - analysis of attributes than fifth grade students in Chapter 1 . 
There was no significant difference between the treatment groups at the fourth grade or between 







males and females. 
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This seems to be a skill the HOTS program is successful in raising. The programs effects 
for analysis of attributes are greater after two years of the treatment supporting the need for two 
years of participation in the program. The results of the interviews with students do not give 
much insight into research hypotheses three, four, or five. The interviews indicate that students 
are not aware of what skills they are working on in the Higher Order Thinking Skills program. 
When asked what they were learning students responses generally included references to 
strategies. The sttidents were able to give examples of strategies they used with the computer 
software, but were unable to generalize to the regular classroom. These findings from the 
interviews are in line with the HOTS program's beliefs about not teaching the thinking skills 
directly to students, but may indicate a need for more coordination between the regular classroom 
and the HOTS program. 

Summary 

These finding suggest that the Higher Order Thinking Skills program is efiective in raising 
students' self-concepts and some higher order thinking skills in the fifth grade. The findings 
suggests that the program is more effective after two years of treatment, and that females in grade 
5 are eflfected more than males. Both the HOTS program and the Chapter 1 program raised 
students achievement scores. However, this study failed to find any statistically significant 
difference in achievement of students in the HOTS program as compared to students in Chapter 1 
programs. It is not certain what other thinking skills the HOTS program might affect or if these 
students' achievement scores will go up later as a result of this treatment. This study's findings 
also suggest that females in these grades may be developmentally more ready for this treatment 
than males. 

These results, along with the interviews and observations, indicate that there needs to be 
more coordination between the HOTS program and the regular classroom. Classroom teachers 
need to be trained to use the questioning and probing techniques used in the HOTS program and 
provide more tie in between the HOTS strategies and the regular classroom. Students stated tha», 
even though they did not like it when they were being questioned or when they were struggling to 





work out an answer to a difficult problem, they felt better about themselves and felt they learned 
more than they did in the regular classroom. 

Is the HOTS program effective enough to warrant the expense? If schools are interested 
in raising students' self-concepts, this program seems to be effective, but more research is needed 
before the program can be evaluated for its ability to raise achievement and higher order thinking 
skills. 

Recommendation.s for Future Research 

The results of this study suggest many directions for future research. Some questions 
which might be answered by future research include the following: 

1. What other types of higher order thinking skills does the HOTS program raise? 

2. Would students' achievement scores increase significantly if the treatment were begun 
earlier and continued for more years. 

3. Would training regular classroom teachers in HOTS techniques help to raise 
students' achievement scores? 

4. Would more coordination between the HOTS program and the regular 
classroom raise students' scores? 

5. Would training Chapter 1 teachers to use HOTS techniques raise Chapter 1 
students' scores? 

6. Would the HOTS program be effective without the use of computers? 

7. Would the gap between male and female students be closed by using the HOTS 
program in later grades? 

8. Would different measuring tools result in similar findings? 

Additional research should be conducted to determine if the Higher Order Thinking Skills 
program is effective. If so, what parts of the program work and how can they be adapted to be 
most effective. If schools are to make informed decisions about how best to help their students, 
educational progi'ams must be evaluated to determine their merit. More research is needed before 
conclusions can be drawn on the Higher Order Thinking Skills program. 
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